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ABSTRACT 

These two papers were presented at a symposium on 
instructional technology in human resource development (HRD) 
moderated by Mark Lewis at the 1996 conference of the Academy of 
Human Resource Development. "An Analysis of the Instructional 
Technology Competencies Required by HRD Practitioners and Sources of 
Competency Development" (Julie A. Furst-Bowe) describes the 
computer-based technologies and distance learning systems currently 
used in employee training and technologies that will be used in 
future training processes. Following an examination of the 
competencies needed by trainers to use the various types of 
technologies, the study identified where trainers are obtaining 
competencies in the use of computer-based technologies and distance 
learning and the barriers to implementation. "Electronic Performance 
Support for Telephone Operators" (Theo J. Bastiaens et al.) examines 
the effectiveness of electronic performance support systems (EPSS) 
and describes the analysis, construction, and evaluation of an EPSS. 
Papers contain references. (KC) 
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An Analysis of the Instructional Technology Competencies Required 
by HRD Practitioners and Sources of Competency Development 



Julie A. Furst-Bowe 
University of Wisconsin-Stout 

The purpose of this study was to describe the computer-based technologies and 
distance learning systems currently used in employee training and to determine 
which technologies will be used in future training processes. The competencies 
needed by trainers to use the various types of technologies were also examined. 
The study identified where trainers are obtaining competencies in use of 
computer-based technologies and distance learning systems. The barriers trainers 
face when implementing new technologies were determined. 



Training professionals perform in a great variety of roles as they apply their 
human resource development challenges facing their organizations (McLagan, 1989). Two primary 
duties of trainers include designing and delivering instruction. Each of these duties is becoming 
more challenging as technology evolves. The use of technology in training has grown 
tremendously over the past five years; tools have improved and have produced several c ges m 
the way training is being designed and delivered (Haag. 1993). Contemporary design and delivery 
systems include computer-based training systems, multimedia systems, electronic performance 
support systems and telecommunication systems for distance learning. In addition, computer 

technology is being used to enhance traditional classroom training. 

Although not every organization has implemented these new design and delivery systems, 
the number of organizations using these systems increases each year. A recent survey of 
organizations with more than 100 employees indicated that 48% are using computer-based training. 
27% are using multimedia systems and 43% are using some type of distance learning system 

deliver training (Industry Report, 1995). . 

The use of technology-based delivery systems in the training process has many potential 
benefits for organizations. For example, the use of computer-based technology in the design and 
delivery of training can result in greater learning gains, more consistent and acceptable job 
performance, enhanced cost-effectiveness and greater flexibility regarding the time and locations of 
training (Hannum, 1990). Several research studies have concluded that, under the right 
circumstances, computer-based delivery systems are considerably more cost effective than 
classroom training and produce learning that is at least equal to what can be ac ev ® 
classroom (Haag, 1993). Technology can also help when addressing the needsof geographically- 
dispersed trainees and in reducing the need for classroom facilities (Perlstein, 1993). 

However, the value derived from the use of technology in training is not due to the 
hardware itself but rather to the instructional processes that technology can support (Hannum, 
1990). To be effective in their positions, today’s trainers must possess competencies needed 
perform in an increasingly technological environment They must have a solid understanding o 
learning theories and methodologies and be able to apply this knowledge to the development and 
delivery of training using computer-based technologies, distance learning systems and other types 
of instructional technology (Hannum, 1990). Trainers who lack these skills may be limiting their 
effectiveness and their ability to obtain positions or advance in many areas of the field. 

Developing and maintaining expertise in instructional technology can be a challenge _to 
trainers for many reasons. Because instructional technology is an emerging field, many of the 
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concepts associated with this field, including "multimedia" or "distance learning, have taken on a 
wide range of meanings, resulting in confusion for practitioners (Anglin, 1991). Also, because 
many of these technologies are still evolving, there are few standards in the field. The hardware 
and software associated with these technologies is continually being changed, upgraded or replaced. 
It is often difficult for both new and experienced trainers to design, develop and implement 
hardware and software in a timely manner when the technology environment is dynamic and the 
rate of obsolescence is increasing (Anglin, 1991). Finally, many trainers who have spent several 
years using traditional training methods and media may resist or feel uncomfortable with new 

technology (Schaaf, 1992). . 

There is a need few a better understanding of the competencies required by trainers 
regarding the use of new technologies in training and how these competencies may be developed. 
Although there have been several needs assessments conducted to identify the competencies of 
trainers, including the American Society for Training and Development s Models f or HRD Practice 
and the competency studies published by the International Board of Standards for Training, 
Performance and Instruction, typically, studies of this type do not provide any detailed information 
about the competencies needed to use specific types of hardware, software or delivery systems. 

Currently, there are several resources that could provide training to trainers who need to 
develop their knowledge, skills and abilities in designing and delivering instruction using 
contemporary instructional technology. These resources include undergraduate and graduate courses 
at colleges and universities; courses, workshops and certificate programs offered by trade schools 
or technical colleges; and conferences and seminars offered by professional assoc i ations (Lindstrom, 
1994). Many hardware and software vendors also provide training to organizations that purchase 
their equipment and products. Finally, trainers can train themselves using training courseware, 
computer tutorials, videotapes, books and manuals (Lindstrom, 1994). However, because these 
programs and resources are relatively new, no statistics exist regarding the percentage of trainers 
who complete formal or self-paced programs or how successful these programs are in meeting the 
needs of trainers. 

In addition, even trainers who are highly skilled in the use of instructional technology 
may experience difficulty when implementing new types of delivery systems in the work 
environment There are often barriers in organizations that inhibit the successful implementation 
of instructional technology in training. These barriers may include high costs, lack of 
management support lack of trainer skills, cultural defaults for the classroom experience and 
failure to identify needs adequately (Gery, 1994). 



Statement of the Problem 

Little is known about the nature of the challenges encountered by trainers as they attempt to 
incorporate computer-based technologies and distance learning systems into their training processes 
and programs. Few studies have examined the impact of these technologies on the role of the 
trainer. 



Purpose of the Study 

The purpose of this study is to provide current information on the implementation of instructional 
technology in employee training and the competencies needed by trainers to utilize instructional 
technology in their jobs. This information may be used to assist training professionals in 
determining their continuing education or training needs in the area of instructional technology. 
The information also may be used by universities, professional organizations and others who 
provide degree programs in human resource development or instructional technology in developing 
relevant curricula. 
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Research Questions 

This study sought to find answers to the following questions: 

1 . What types of computer-based technologies and distance learning systems are 

being used to deliver training in business and industry? 

2. what types of computer-based technologies and distance learning systems will be 

used to deliver training in the future? 

3. What are the competencies that are needed for trainers to deliver instruction using 
computer-based technologies and distance learning systems? 

4. Where are trainers obtaining the competencies that are needed to deliver 
instruction using computer-based technologies and distance learning systems? 

5. what barriers exist in the workplace that prevent trainers from using computer- 

based technologies and distance learning systems to deliver training in the 
workplace? 



Significance of the Study 

Technology has dramatically changed the way can be which training is designed and delivered. 
These technological trends necessitate that training professionals learn new job skills (Linds trom, 
1994). This study was designed to benefit training professionals, human resource development 
managers, academicians and other training providers by providing practical, timely information that 
may be used to update trainer skills and training programs. This study will provide current 
information on the implementation of instructional technolop in training efforts at a time when it 

is crucial for trainers to expiand their repertoire of skills in this area. . 

In order to best prepare training professionals with the necessary knowledge, skills and 
competencies, training providers need to have an accurate picture of the current skiU requirements 
(Morlan & Lu, 1994). The findings of this study may be used by universities and other 
organizations to develop and revise degree programs, courses, seminars, workshops, and self-study 
materials to meet the instructional technology training needs of human resource development 

professionals. 



Methodology 

This study was conducted during a six-month period from June to December in 1994 First, a 
literature review focusing on the use of instructional technology in training was conducted. Next, 
a questionnaire was developed by the researcher and reviewed by a group of eight trainers, researcn 
consultants and experts in the field of instructional technology. The first section of the survey was 
to contain demographic items, including job title of respondent, type of organization andsizeot 
organization where the respondent is employed. In the second section, respondents were to identity 
(a) how technology is currently being used to design and deliver training in their organizations, (b) 
their perceptions of the types of technology that will be used to design and deliver training in the 

future (c) the level of competency required of trainers in each type of technology, and (d) sources 

of competency development in each technology In the third section of the survey, respondents 
were asked to identify barriers in the workplace which limited the implementation of instructional 

technology in training. , . f 

Population and Sample. This study was designed to determine the perceptions of 

training professionals regarding the use of technology in the design and delivery ofmstruction. 
The population selected for this study included members of the National Society for Performance 
and Instruction, specifically those members living and/or working in the following Midwestern 
states: Illinois, Iowa, Michigan, Minnesota and Wisconsin. This group was selected over the 
other professional training associations because of its focus on performance technology. The 
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association’s 1993-94 national membership directory was used to obtain a list of members of the 
population for this study. In this directory, members were listed alphabetically by state. In the 
five Midwestern states included in the study, there were a total of 1,093 members. A systematic 
sample of members was selected. This method of sampling is appropriate when a list of elements 
is available and when the list is arranged in a manner that will not interfere with the purpose of the 
study (Babbie, 1990). This method also assured that a proportional number of individuals from 
each of the five states would be included in the sample. 

The following steps were taken to determine the size of the sample. First, a decision was 
made to establish a confidence level of 95% (0.95) for die results of the study. Then the formula 
outlined in How to Determine Appropriate Survey Sample Size (Narins, 1994) was applied to 
calculate the appropriate sample size for the population. For a population of 1,093, a sample of 
381 was required to produce the desired confidence leveL This method of determining sample size 
was generous and provided latitude against typical sources of error including non-response. The 
first member was chosen at random from the list of members in the designated five-state area. 
Then every third member was selected until the sample had reached the desired size. 

Instrument Development. Given the purpose of the study, die research questions to 
be answered and the size of the sample, a mail questionnaire appeared to be the most economical 
and appropriate data collection technique. The instrument was developed through a careful 
examination of similar studies found in the review of literature. The instrument was designed to 
collect data as a self-administered questionnaire. 

The instrument consisted of an 1 1 x 17-inch sheet of paper which was printed on both 
sides and folded into a four-page booklet The body of the questionnaire was divided into three 
sections. In the first section, respondents were asked if their current position involved designing, 
delivering or managing training. At this point, individuals who were not currently working in the 
field of training and development were instructed to send back the survey without answering the 
remaining survey items. Respondents who were currently employed in the field were instructed to 
provide other demographic information, including their job title and the size and type of their 
organization, and to complete the remaining three pages of the survey. 

The second and third pages of the survey contained a grid that listed 32 types of 
instructional technologies divided into categories. These categories included computer-based 
training systems, multimedia systems, electronic performance support systems, virtual reality, 
distance learning systems and computer presentation systems. Additionally, five areas were further 
divided into subcategories. In each category and subcategory, respondents were asked to identify if 
they used each technology in their training efforts and if they planned to use each technology in the 
next three years. They were also provided with an option to list additional technologies which 
were not included on the questionnaire. 

In this same section, respondents were asked to assume that their organization planned to 
use each technology and to identify the levels of competency that would be needed to implement 
the technology. The levels included the ability to use or assist trainees in the use of a particular 
technology, the ability to assess the effectiveness of a technology, the ability to select a 
technology for an organization and the ability to develop a program or system using the 
technology. Respondents were allowed to select as many levels as they felt were appropriate. 

Respondents were then asked to identify where they obtained or would plan to obtain 
competency in each technology. A list of training sources, including colleges and universities, 
technical colleges, seminars and conferences, vendor-sponsored training and self-study methods, 
was provided. 

In the third section, printed on the fourth page of the survey, respondents were asked to 
identify barriers in the workplace which limited the use of instructional technology in training. In 
this section, respondents were provided with a list of potential barriers and were asked to indicate if 
the barriers were present in their work environment These barriers included insufficient funding; 
hardware incompatibility; lack of management interest or support; lack of time, knowledge or 
technical skills among trainers; lack of interest among trainees; inadequate needs assessment; and 
lack of technical support Respondents could also identify additional barrios if desired. 
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A cover letter was developed, printed and mailed with each survey. The cover letter 
explained the purpose of the study and the format of the questionnaire. It also emphasized the 
importance of respondent input and provided the name and complete address of the researcher. The 
cover letter also contained the informed consent information required by the researcher’s university. 

Data Collection. The 381 questionnaires were mailed with cover letters and postage- 
paid return envelopes on October 7, 1994. A pencil was included as incentive for individuals to 
respond to the survey. Seven of the surveys were returned to the sender due to an incorrect or 

address. These individuals were removed from the sample, reducing the sample size to 
374. Thirty percent of the sample, 112 individuals, returned the survey by November 5, 994. 
On November 6, 1994, a second mailing, which included the questionnaire, a second cover letter, a 
postage-paid envelope, and a packet of instant coffee, was mailed to the remainder of the sample. 
Bv December 12, 1994, 49% of the sample, 184 individuals, had returned die survey. A phone 
follow-up was conducted between December 12 and December 16, 1994. An attempt was made to 
phone every individual who had not returned the survey. During the phone calls, individuals rwere 
asked several questions related to the use of instructional technology in training and reminded to 

send back the original survey. . . 

As a result of the phone interviews, 54 additional people were removed from the samp 
because they were no longer employed at the organization listed in the directory. Thus, the final 
sample size was determined to be 320. Following the phone calls, another 15 surveys were 
returned. The final number of responses was 199, for a response rate of 62%. Of the 199 
individuals who returned the survey, 52 individuals indicated that they were not involved ui the 
design or delivery of employee training programs (mostly university faculty members, students and 
retirees) and did not complete all of the items on the survey. This left a total of 147 completed 

aiiestionnaires to be analyzed. TT . . 

Data Analysis. On December 23, 1994, the surveys were delivered to the University 

of Wisconsin-Stout’s Academic Computer Center to be analyzed. The data were tabulated, and the 
surveys were returned to the researcher on February 3, 1995. In all sections of the survey, each 
item was analyzed in terms of frequency of each response and overall percentage for each option 
provided. In addition, Z-tests on the difference of proportions between the yes responses of 
current users and the "yes" responses of those who planned to use each technology were conducted 
to determine if there were significant differences between current and planned future usage. Chi- 
square tests were conducted to determine if there were significant differences between the various 
training sources selected by respondents. 



Results 

The results of the survey indicated that organizations are currently using a Wide range of 
technologies, and there are 12 technologies that are currently being used by at least 50% of the 
respondents. These technologies include computer-based training, computer tutorials, computer 
simulations, computer presentation systems, presentation software, electronic perfwmance support 
systems, on-line help systems, information databases, multimedia systems, LCD panels, LCD 

v ideo/ data projectors and local area networks. 

These were the same technologies that 50% or more of the respondents indicated they plan 
to use in the next three years. However, respondents indicated that they plan to make significantly 
greater use of multimedia development and deliveiy tools including authoring programs, interactive 
video CD-ROM, compact disk interactive and digital video interactive. They also indicate that 
they plan to use more complex technologies for their computer-based training and electronic 
performance support systems, including hypertext, expert systems, embedded/concurrent training, 
intelligent tutoring and virtual reality. There will also be greater use of computer conferencing to 

deliver training over distances. . .. , 

According to respondents, there will be less use of certain types of distance learning 

systems, such as audioconferencing and one-way video. This would be logical as technology 
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continues to evolve and provides more advanced, interactive systems for the delivery of distance 
education (Gery, 1994). There will also be a decrease in the use of computer presentation systems 
to deliver classroom training. This decline will occur as companies abandon classroom training for 
more effective and cost efficient electronic instruction delivered at the desktop (Galagan, 1994). 

The levels of competency required to implement instructional technology in training 
pro grams were consistent across 27 of the 32 types of technologies included in the study. Overall, 
respondents reported that the ability to use or assist trainees in the use of the technology was the 
most highly needed competency. The ability to evaluate the effectiveness of a specific technology 
was also frequently identified. The ability to develop programs or systems was identified by few 
respondents as being necessary for most technologies. 

Vendor- sponsored training and self-study methods proved to be die most popular choices 
for developing competency in instructional technology. Attending seminars, conferences or 
workshops was frequently identified for developing competency in some technologies; however, it 
was not the primary method for competency development in any specific technology. 
Significantly fewer individuals indicated that they would attend courses and programs at 
universities, four-year colleges or technical colleges to develop their skills in any of the areas 
included in the questionnaire. 

The respondents indicated that a lack of time and a lack of financial resources are the 
major barriers in implementing instructional technology in training efforts, as these barriers were 
cited by approximately 75% of the respondents. Lack of compatibility between systems, lack of 
management support, lack of technical support and lack of trainer skills are barriers that were 
identified by more than 50% of the respondents. These findings were consistent with the literature 
on this topic, which indicates there are several reasons why technology has not been fully 
integrated into training programs, including high costs, lack of management support and lack of 
skills among trainers (Gery, 1994). 



Conclusions 

From the findings of this study, it can be concluded that the major types of instructional 
technologies used in training and development will not change dramatically over the next three 
years. However, there are several newer, more sophisticated technologies that will be used with 
greater frequency in the future. The respondents’ current and planned uses of instructional 
technology are consistent with other recent studies on this topic that indicate there will be a greater 
use of interactive technologies that will change how, when, and where trainees learn (American 
Society for Training and Development, 1994). It is predicted that in the future more companies 
will utilize digital multimedia technologies and individualized performance support systems to 
provide flexible training opportunities to workers (Galagan, 1994). 

From the survey responses, it can be concluded that it is far more important for trainers to 
be able to use and evaluate new technologies than to be able to design and develop their own 
programs or systems. The data from the survey support the concepts found in the literature in this 
area. Past studies on this topic have concluded that trainers should be familiar with the 
applications of instructional technology; however, program or system development is generally 
done by computer programmers or media specialists with expertise in these areas rather than by 
trainers themselves (Spitzer, 1988). 

It was determined by the respondents that vendor-sponsored training and self-study 
methods are the primary sources of competency development in instructional technology. 
Seminars, conferences and other training programs sponsored by professional organizations appear 
to play a lesser role, and universities and technical colleges appear to play a minimal role in 
providing trainers with knowledge and skills in computer-based training, multimedia systems, 
EPSS, distance learning systems or computer presentation systems. 

Finally, it can be concluded that a lack of time and a lack of financial resources are the 
major barriers to implementing instructional technology in training efforts. Lack of compatibility 
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between systems, lack of management support, lack of technical support and lack of trainer skills 
are also significant barriers. However, there does not appear to be a lack of trainee interest in 
using instructional technologies or a general lack of support for training efforts. 



Recommendations 

This study was designed to assist training professionals, human resource development managers, 
academicians and others who offer training and degree programs by providing information on how 
instructional technology is currently being used in training and how it may be used in the future. 
These various groups may find the results of this study useful in future planning efforts. 

Recommendations for Practitioners . It is recommended that training 

professionals and their managers use the data generated by this study in defining current and future 
training needs and in identifying resources to obtain new skills and competencies in instructional 
technology. In particular, training professionals should become familiar with die technologies mat 
are currently being used by more than 50% of the respondents’ organizations. Training 
professionals, whether working in small, mid-sized, or large organizations, should also become 
knowledgeable regarding digital technologies and other emerging technologies that large numbers 
of organizations plan to use in the future. Skills in using computer systems and electronic 
support sy stems have been formally recognized as essential competencies for training professionals 
(McLagan, 1989). The shift from face-to-face training to delivering information during the 
performance of work will require all trainers to become familiar with numerous delivery 

technologies (Galagan, 1994). , . 

The data from this study suggest that training professionals should focus on developing 
competency in the use and evaluation of the various technologies included in the survey. 
However, in a small number of organizations, it is also required that trainers select and develop 
programs and systems. In addition to learning about hardware and software, comments from 
respondents and the literature indicate that trainers also should possess competencies in using 
traditional media technologies. Trainers should also understand the process of applying appropriate 
instructional technology to performance problems (Piskunch, 1993). 

Recommendations for Training Providers. The results of this study may also 
assist faculty in colleges, universities and technical colleges; directors of professional 
organizations; hardware and software vendors and others who provide courses, programs and 
training in the field of instructional technology. The majority of respondents in this study 
preferred vendor-sponsored training and self-study methods to meet their instructional technology 
training needs. However, several respondents indicated that they and their colleagues have not been 
trained in many aspects of instructional technology and lack the knowledge or skills necessary to 
be effective in this area, therefore, it is recommended that venders and organizations that provide 
self-study materials expand their offerings in instructional technology training, particularly in the 
area of emerging technologies to help address this unmet training need. 

Although several post-secondary institutions offer programs in instructional technology, 
few trainers seem to be taking advantage of these offerings. It is recommended that undergraduate 
and graduate programs in training and development and instructional technology review their 
instructional technology courses and competencies in light of the findings of this study to ensure 
that their courses are relevant and appropriate for training professionals. 

Due to i nadeq uate budgets for equipment and laboratories, it is often difficult for colleges 
and universities to stay current with technology (Lindstrom, 1994). However, if universitiM and 
tech nical colleges are truly interested in meeting the instructional technology training needs ot 
training professionals, it is recommended that they form partnerships with vendors or professional 
associarinne to assist in offsetting the costs of providing hardware and software training on college 
campuses. 
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Electronic Performance Support For Telephone Operators 

TheoJ. Bastiaens 
WimJ. Nijhof 
HarmenJ.Abma 

University ofTwente, The Netherlands 

This research is about die effectiveness of Electronic performance support 
systems (EPSS). Some of the assumptions related to EPSS are evaluated This 
paper describes the analysis, construction and evaluation of a performance 
support system for telephone operators. From the research some conclusions can 
be drawn that may be important for other projects in the field of Electronic 
Performance Support 



Since the large scale introduction of computers in the early eighties a lot has changed. In the field 
of performance technology several electronic support systems are introduced. Nowadays 
keywords are "just in time training" and "learning by doing". Computer support and electronic 
job-aids are approximating the master in the master and journeyman relationship used in 
eattisr days. This article is a report of a project in the field of Electronic Performance Support 
Systems (EPSS). Firstly the focus is on "What is an EPSS?". Secondly the theoretical advantages 
of EPSS use are stated? At last findings of a research project are discussed. In the project an 
Electronic Performance Support System for telephone operators is optimised and evaluated. 

An EPSS is an integrated computerised environment that supports and occasionally 
monitors employees while they perform their jobs. In general an EPSS contains the following 
four components (see figure 1): tools (to perform the job), information (needed to do the job 
correctly), advice (for the difficult parts of the job) and training (to extend the employees’ 
knowledge and skills). It substitutes for or enhances the support of a master. 




In literature several possible advantages of the use of an EPSS are reported. These are 
assumptions of possible advantages of EPSS use in practice. The first advantage relates to on-the- 
job training which leads to high transfer, no need to leave the wo rkpl a c e, and more-active 
learning processes (Bastiaens, 1995). Probably the most important advantage is the immediat e 
access to information, training and advice (Gery 1989, 1991). The just-in-time access to 
information leads to an extension of the employees' long-term memory and a reduction of the 
working load memory (Law, 1994). Having continuous access to training means a reduction of 
formal tr aining in advance of task performance. Because employees can constantly consult the 
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advice part of the EPSS the need for supervisor's guidance is expected to be less. This has the 
additional advantage that the responsibility focus shifts from trainer and training program to the 
individual’s learning needs (Geiy, 1991). Moreover, performance support can be important for 
employees self-management of for the guidance of self-directed teams (Bramer & Senbatta, 1993) 
and such has the potential to improve the worker's productivity (Raybould, 1990; 1991). 



Problem Statement 

In the previous section assumptions on the advantage of EPSS have been mentioned but little has 
been empirically proven. This research project attempts to evaluate the effectiveness of an EPSS. 
A few disadvantages were found in the literature that could affect the effectiveness of an EPSS. 
These disadvantages are split in to three categories. 

The first category is related to the teaming process. "Just-in time" training at the 
workplace, providing employees with small task-oriented training granules and employees taking 
control of their own learning process can create problems. Clark (1992) argues that employees 
may fail to build a unified picture of their job when they have to extract information from an 
EPSS. Several small information parts will create a fragmented knowledge base. Novices 
especially need a high level overview of the content to relate details of training. Clark doubts 
about the learner control in EPSS and she illustrates it with research of Milheim and Martin 
(1991) which indeed proves that learner control is not as effective as instructional control 

The second category involves problems related to innovation. It is expected that the 
introduction and implementation of EPSS will summon resistance. Employees are not likely to 
give up working the old way*. Even if they are willing to try a new method there is the problem 
of pressure in their work. Employees will simply not have the time to engage the training 
support 

The third category is related to support and work. Is the support adequate? Will "just-in 
time* support ’de-skill’ workers? Will it ’demotivate’ workers? Or will it automate the low level 
tasks and bring in more time to perform tasks on a higher level (Carr, 1992)? 

Within the framework of this study it is impossible to give answers to all the questions. The 
research is therefore restricted to the questions stated in the section ‘research questions’. 



Research questions 

The main purpose of the research is to evaluate the developed support environment and 
optimise it The main question is as follows: What is the effect of the existing support 
environment on the performance and learning of the telephone operators? 

Further the following sub-questions are distinguished: 

How can an optimisation of the existing support environment further improve the effectiveness of 
the support environment? 

After the optimisation the effects are evaluated on improvement So the third research 
question is: "What is the effect of the optimisation on the performance of the telephone 
operators?" 

Regarding the research questions we have the following hypotheses: 

Hypothesis related to differences between novice and experienced operators: 

There is a difference between novice and experienced operators. Novice operators need more and 
other information than experts. 

Hypothesis related to the support of the information component It is expected that 
novice operators will appreciate the information component more, because of a more serious need 
for information. Lack of information will affect their performance. 
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Hypothesis three related to the motivation of the operators: Novice operators have a 
higher motivation but are more insecure about their own performance than experienced 
operators. For that it is expected that novices have a higher score on innovation willingness. 

Hypothesis related to the treatment: It is expected that after the optimisation the new 
information component is used more often. It is also expected that a new information component 
will reduce the use of other information sources. At last the assumptions and disadvantages stated 
in literature are compared with the effects related to learning, innovation, support and work in 
this research. 



The Setting 

The research project was executed in co-operation with a large Dutch banking organisation. They 
only provide banking services and advice by telephone. For that their telephone operators are 
extensively trained. As far as the training and experience, two types of telephone operators are 
distinguished. Firstly, experienced operators who have the skills to give advice for all the 
products the bank has to offer to their clients. Secondly, novice operators starting with just a few 
products. Novice operators combine training and working and extend their knowledge and skills 
during a period of six months. After six months training the novices’ workers can give advice for 

all the products. , . . A A . 

To give advice and information to clients and also to process their clients data the 

telephone operators are supported by a personal computer that is attached to a mainframe. Each 
operator has a large screen monitor at his or her disposal. On this monitor it is possible to use a 
tool environment (to process the data) and an information environment (quick access to up-to- 
date information) simultaneously. 



Methodology 

To evaluate the support environment the methodology of the one group pre-test post-test design is 
used (figure 2). This design exists of a pre-test, a treatment and a post-test For practical reasons 
it is not possible to use a control group. 

The pre-test provides an insight in the information use of experienced and novice 
operators. To establish an improvement after the treatment a comparison of the results of the pre- 
test with the results of the post-test are made. For that the post-test is a replica of the pre-test To 
improve the reliability of the findings the method of triangulation is used (Patton, 1990). 

The population consists of 100 employees of the sales and service department savings- 
accou nt* The variety in working experience, age and educational background is checked and the 
test groups are checked for homogeneity. 



Figure 2: Overview of the methodology. 
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The Theoretical Construct 

Theoretical constructs have been extracted from an identical research project that was carried out 
earlier (Bastiaens, Nijhof Abma, 1995). The constructs enclose the variables that exert an 
influence on the EPSS (table 1). Working, treatment and background are general variables. The 
constructs' tool and information together are the support environment Next to this, attitude 
towards work and performance are important, to measure the influence of the context To get an 
insight of the knowledge, skills and attitudes learned in the introductory course some variables 
are identified and taken into consideration. The course may influence the use of the support 
environment 

From the constructs variables are derived from which items are formulated. These items 
were used in questionnaires and given to the operators. 



Table 1: A list of constructs 



Constructs 


Variables 


Woric 

attitude towards work, 
performance 


motivation, sdf-oonfidence 
independence 


Tool 

oomnnmi cation 

help 

menu 


interface, technical construct, technical realization 
content, cornmunkatioo 
service and advice 


Information 

communication 

information 

information (other origin) 


interface, technical construct, technical realization 
service and advice, usefulness, structure 
information 


Course 

preparation on performance 


Knowledge, skills; attitude 


Background 

Personal 

Experience 

attitude towards innovation 


age, sex 

educational background, working and computer 
working with new technology 



Results on the Pre-test 

The pre-test started with semi-structured interviews (n=8). From the interviews the researchers 
wanted to learn more about the work conditions in the section, working with the tool, the use of 
information and the use of the information part in the support system. It is not possible to go into 
detail in this paper so in short; The interviews showed that the operators in general were very 
satisfied with their job. They liked their jobs and the tool. About the information part in the 
support system they were not so satisfied. That resulted in a low average use of the information 
component caused by obsolete detailed information and a dull interface with to many levels in a 
hierarchical structure. It was easier to consult a colleague or to use your own notes (which were 
used very often as little job aids). 

The second instrument of the pre-test were the observations. In this part an observer sat 
next to the telephone operator when he/she was working. The most important variables used in 
this measurement are conversation time, information use and the nature of the conversation 
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(question, problem, complaint). The independent variables were gender and experience. Exactly 
150 conversations were observed. In 40 cases the operator used an information source. Table 2 
shows us the nature of the conversations and the use of information. The general conclusion 
derived from table 2 is that the information component in the support system is not used very 
often. The operators use more often another source. A closer look at that sources showed that 
they adfftd their colleagues for information 8 times, used written sources as folders, brochures, 
handbooks etc. for 28 times (not in table). 

Regarding the first hypothesis which is related to differences between novices and 
experienced operators the observations showed a difference between novice operators and 
fy jyrirnrai operators. Not only in the information use, as expected experienced employees do 
need less information, but also in conversation time, the average time for experienced operators 
is 2.31 minutes (s.d. 2.07), for novices 3.08 minutes (s.d. 2.40). 

The third instrument was the questionnaire. In the questionnaire 24 operators were asked about 
their motivation, work, tool and information use. First their motivation was looked at. 

Our hypothesis that there is a difference in motivation between novice (nl) and 
experienced operators (n2) is not true (Mann- Whitney test, nl= 8, n2= 15, U = 55.0, p= .78, 
double tailed). Novices do not have a higher motivation related to the work. 

The hypothesis, novices are more insecure than experienced operators who are more 
satisfied about their performance is not true(Mann-Whitney test, nl= 8, n2 = 15, U= 53.0, p= 
.34, one tailed) Both types of operators value their performance as high. 

The hypothesis that novices score higher on innovation willingness is also not true. Both 
groups score also the same on innovation willingness (Mann- Whitney test, nl=8, n2= 16, U= 
63.5, p= .97). There is no difference in how they look towards new technology. 

Important for the optimisation they were asked about the technical realisation of the 
tool. When the telephone operators were asked their opinion about the tool it showed that that the 
two groups had the same opinion about the technical realisation (errors in software, waiting time, 
etc.), the help content (procedural information) and the communication with the help. Table 3 
shows the results. 

The following results are related to the support of the information component The 
operators were asked about the technical realisation, the information accuracy and the use of 
other information sources. Table 4 shows no significant difference between the two groups. 

There was one important difference (not in table 4); experienced operators use the information 
component more often and for more ends (Mann- Whitney test, nl= 8, n2= 16, Lb* 26.5, p= .019, 
double tailed). The hypothesis that novices use the information component more often than 
experienced operators is not true. 

In general the following can be concluded. For most hypotheses the pre-test shows no 
difference between experienced and novice users. It is a surprise that experienced operators use 
the information component more and for more ends than a novice All the employees told some 
impnxaing information about how to improve the system. A new support system was constructed 



Recommendations for the Construction of a New Support System 

In general the findings show that the operators are satisfied about the tool. A few administrative 
improvements are suggested. The real improvement has to be made in the information 
co m ponent The data shows that experienced users actually do use the information component 
more often. They use it as a reference bode simply because not every detail of a product is 
remembered. On the other hand novices use other information sources more often, even for 
factual knowledge (where experienced users use the system). It appears that novices do not know 
how to use the system because they are confused because of the hierarchical design. For them, it 
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is hard enough to advise a client and operate the telephone system without using an information 
system that is not user friendly. 

It is also noticed that they need an other sort of information. Not only knowledge of the 
facts but also instructions about how to do the job. This sort of procedural information was not 
available in the system yet It shows that it is also important that users can rely on the 
information. Information has to be up-to-date and complete. The last important recommendation 
is that the search and use of the information component has to be an integral part of the overall 
performance. That means the operators have to integrate the information search as a skill in their 
performance. 

The recommendations require a context- and a task- analysis before a new information 
component can be constructed. Also some technical conditions have to be taken into a rco»"t 
Figure 2 shows us the schematic design of the traditional information component Figure 3 shows 
us the constructed new information system with a maximum of three layers and help questions 
(Bastiaens, 1994). 



Figure 2, Figure 3. 



Products 




Savings (general) 


• giro savings 




• automatically What? How? 
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Various 




* address and phone 



Results on the Post-test 

The post-test consisted of the same instruments and identical variables as were used for the pre- 
test, so semi-structured interviews (n=8) were used first. As in the pre-test the operators are in 
general very satisfied with their work. One important aspect they told the researchers the second 
time was the deficiency of standardisation in the work. Although the system provides standards 
to process the data, operators want more uniform procedures. The operators are also less positive 
about the communication process in the division. New rules, products and procedures are not 
communicated as quick as the should be. About the new information component they are very 
satisfied. They use the information component because of the new user-friendly structure, the up- 
to-date information. On the other hand for lack of time on the workplace they want to explore the 
component off the job. They miss a personal scratch-pad in the system and the possibility to 
structure the interface themselves. 

The next part of the post-test were the observations. Again 150 calls were observed. 
Table 1 shows the results. It shows that the information system is used 11 timre other 
information sources 26 times. The total use of information is 37 times. When a closer lode is 
taken at the data it shows that the operators consult colleagues for 9 times and use brochures and 
handbooks for 16 times (not in table). The observations in the post-test showed us again a 
difference between novice operators and experienced operators. Not only in the information use, 
as expected experienced employees do need less information, but also in conversation Hnw» the 
average time for experienced operators is 2.19 minutes (s.d. 1.48 ), for novices 2.25 minutes (s.d. 
2.19). 

The last part of the post-test was a questionnaire. The questionnaire was filled out by 27 
operators. Table 2 shows the results of a comparison between experienced en novices. No 
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significant diKetmx was found when the operators were asked their opinion about the tool again 
(technical realisation, the content of the help and the communication with the help component). 

The result related to the information component show that their opinion about the 
t~-hnw realisation was significantly different The accuracy of the information in the 
information component is appreciated more by experienced operators (table 3). The hypothesis 
re lated to the support of the information component is not true. It was expected that novice 
operators would appreciate the information component more, because of a more serious need for 
information. 

In the use of other information sources is no significant difference between the two groups. 



Comparison Between Pre-test and Post-test 

After the development of a new information component the hypothesis is that the 
information component is used more often than in the old situation. That is true. The 
information component is used more often (ChiMest, df= 1, p= .03). Especially novices us 
information component more often. Their score on the pre-test was 0, after the treatment o: 
post-test their score was 7. This is a 
significant difference (Chi 3 -test, df= 1, p= .70). 

Table 2. Observed use of information in the pre-test and post-test 



nature of conversation 


use of information in 
support system 
pre-test / post-test 


use of other information sources 
pre-test / post-test 


pre-test / post-test 


question 

problem 

complaint 


2 (13%) / 10(6.7%) 
1(0.7%) / 0( 0%) 
1 (0.7%) / 1 (0.7%) 


24(16%) / 19(1X7%) 
1 (0.7%) / 1 ( 0.7%) 
11(73%)/ 6(4.0%) 


26(173%) / 29(23.0%) 
2(13%)/ 1(0.7%) 
12(8.0%)/ 7(4.7%) 




4(2.7%) / 11 (73%) 


36(24%) / 26(173%) 


40 (26.7%) / 37 (24.7%) 



Table 3. Opinion about the tool component in the pre-test and post-test 




* double tailed 
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Table 4. Opinion about the information component in the pre-test and post-test 





pre-test n 


pre-test 

U 


pre-test 

P* 


post-test 

n 


post-test 

U 


post-test 

_e! 


technical 














realisation 




31.0 


.45 




68.0 


.42 


experienced 


8 






13 






novices 


16 






13 






accuracy 














information 


- 


44.3 


J24 




56.0 


.15 


experienced 


8 






13 






novices 


16 






13 






use other 














information 




45.0 


26 




56.0 


.15 


sources 














experienced 


8 






13 






novices 


16 






13 







ft is also an expectation that the new information component reduces the search for and 
use of other information sources. That is not true. In feet especially the consultation of colleagues 
has grown. Fortunately this is not a significant growth (ChF-test, df= 1, p= .06). On the other 
hand the use of written information sources did significantly drop (ChiMest, df= 1, p=-01). 

Our hypothesis that the new information component supports the operators better than 
the old is not true. There is a difference in the meanscoies in advantage of the new component 
but it is not significant (Mann- Whitney test, nl= 24, n2= 26, U= 250.5, p=.12, one tailed). 
Although the users think that the new support is an improvement this is not seen in the 
comparison between the pre-test and post-test 

When an evaluation is made on the theoretical assumptions the projects shows that the 
support environment leads to training on the job. The reduction of formal learning is high. In this 
organisation formal classroom training is reduced from one month to 6 days. The support 
environment also leads to a reduction of the working load memory. The assumption of resistance 
is not observed but it shows that it is hard for the operators to give up working the old way. The 
automation of tasks and the easy consultation of the support environment lead to an extension of 
tacif< But the results show that it is ‘detnotivating , when operators have to wait too long before 
they can broaden their knowledge about other products. 

Conclusions 

The results show that the use of the information component at first was very low. From that a 
lesson can be learned that an information component has to be up-to-date, complete and that the 
use has to be an integral part of the performance. Our advice was to hire one person to update the 
information on a daily base. This person does not only put in official information but also collects 
notes and small job aids which the operators use to develop. Another advice is the integration of 
die use of the support system in the formal learning process. Teach new employees to work with 
the support environment. Give them the tune to explore the support environment before sending 
them to work. 

Before constructing a support system developers have to keep in mind that the people 
who use the system are very diverse. These people have their own needs. Give them the ability to 
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change the interface their way, integrate a notepad and provide different ways to search for 

information. . . ... - 

Although it is no complete EPSS as is stated in the first section, the researchers think 

that the developed support system in this situation meets the needs of the company. In their 
roinion it is not necessary to develop a complete EPSS in every situation. In this rosearc project 
some advantages of electronic support have been seen but also some senous disadvantages. 
Further research in this field has to be done to gather insights not only in the construction of 
EPSS but more important in the analysis of the organisation, the performance and the workers. 
However the researchers think that performance support will be the future, it will inform and 
train employees and help them to do a better job in less time. 
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